^  ‘‘5  :*.i 

^  “V*  ? 


GAO 


United  States 

General  Accounting  Office 

Washington,  D.C.  20548 


National  Security  and 
International  Affairs  Division 

B-259204 

April  19, 1995 

Congressional  Committees 


Accesion  For 

NTIS  CRA&I 
DTIC  TAB 
Unannounced  □ 
Justification 


By . . 

Distribution/ 


Availability  Codes 


Dist 


Avail  and  /  or 
Special 


As  called  for  by  the  Senate  Armed  Services  Committee  report  on  the 
National  Defense  Authorization  Act  for  Fiscal  Year  1995,  we  assessed  the 
concurrency  between  the  development  and  production  phases  of  the  Air 
Force’s  F-22  fighter  program  and  the  risk  associated  with  that 


concurrency. 


Background 


Concurrency  is  broadly  defined  as  the  overlap  between  development  and 
production  of  a  system.  The  stated  rationale  for  concurrency  is  to 
introduce  systems  in  a  more  timely  manner  or  to  fulfill  an  urgent  need,  to 
avoid  technology  obsolescence  and/or  to  maintain  an  efficient  industrial 
development/production  work  force.  For  measuring  the  degree  of 
concurrency  in  this  report  we  used  a  statutorily  required  guide  issued  by 
the  Department  of  Defense  (dod)  in  April  1990  for  assessing  concurrency 
and  associated  risk  in  mqjor  acquisition  programs.  Its  measure  of 
concurrency  is  the  amount  of  initial  operational  testing  and  evaluation 
(iot&e)  completed  before  entering  production  of  a  system. 

Initial  operational  tests  are  field  tests  intended  to  demonstrate  a  system’s 
effectiveness  and  suitability  for  military  use.  iot&e  is  a  key  internal  control 
to  ensure  that  decisionmakers  have  objective  information  available  on  a 
weapon  system’s  performance  and  to  minimize  risks  of  procuring  costly 
and  ineffective  systems. 

In  the  late  1980s,  the  Congress  found  that  dod  was  acquiring  a  large 
portion  of  total  program  quantities,  using  the  low-rate  initial  production 
(trap)  concept,  without  successfully  completing  iot&e.  As  a  result, 
legislation  was  enacted  in  1989  to  limit  lrip  quantities  for  mqjor  systems. 
The  law,  10  U.S.C.  2400,  defined  lrip  as  the  minimum  production  quantity 
needed  to  provide  production  representative  articles  for  iot&e,  establish 
an  initial  production  base,  and  permit  an  orderly  increase  in  the 
production  rate  sufficient  to  lead  to  full-rate  production  after  completion 
of  IOT&E. 


In  the  conference  report  supporting  the  National  Defense  Authorization 
Act  for  Fiscal  Years  1990  and  1991  (P.L.  101-189),  the  conferees  indicated 
that  LRIP  quantities  should  not  total  a  significant  percentage  of  a  total 
planned  procurement.  Later,  the  Federal  Acquisition  Streamlining  Act  of 
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Results  in  Brief 


1994  prescribed  new  controls  for  lrip.  The  act  states  that  the  Secretary  of 
Defense  must  specifically  explain  to  the  Congress  why  any  planned  lrip 
quantities  exceed  10  percent  of  a  planned  production  quantity  of  a  system, 
as  defined  at  the  milestone  II  or  development  decision.  This  provision, 
however,  was  not  in  effect  when  the  F-22  program  reached  milestone  II. 

The  F-22  passed  milestone  n  in  1991.  At  that  time,  the  Air  Force  planned  to 
acquire  648  F-22  operational  aircraft  at  a  cost  of  $86.6  bUhon.  After  the 
Bottom  Up  Review,  completed  by  dod  in  September  1993,  the  planned 
quantity  of  F-22s  was  reduced  to  442  at  an  estimated  cost  of  $71.6  billion. 

We  recently  reported  that  aircraft  systems,  including  the  T45  trainer 
aircraft,  B-IB  bomber,  and  the  C-17  cargo  aircraft,  as  well  as  many  other 
smaller  systems,  entered  lrip  before  successfully  completing  any  iot&e.^ 
This  resulted  in  the  purchase  of  systems  requiring  significant  and 
sometimes  costly  modifications  to  achieve  satisfactory  performance, 
acceptance  of  less  capable  systems  than  planned,  and  in  some  cases 
deployment  of  substandard  systems  to  combat  forces. 

The  lrip  contract  award  is  scheduled  for  September  1997.  lrip  aircraft  are 
those  to  be  procured  during  the  period  of  concurrency. 


Although  the  F-22  program  involves  considerable  risk  because  it  embodies 
important  technological  advances  that  are  critical  to  its  operational 
success,  the  F-22  program  exhibits  a  high  degree  of  concurrency  because 
the  program  will  enter  production  well  before  commencement  of  lOX&E. 
This  concurrency  will  permit  procurement  of  a  significant  quantity  of 
F-22s  before  many  of  the  technology  advances  are  flight  tested  and  before 
completion  of  iot&e.  Historically,  there  have  been  numerous  examples  of 
the  adverse  consequences  of  concurrent  development  and  production,  that 
is,  buying  weapon  systems  before  they  demonstrate,  through  testing,  that 
they  perform  as  required. 

The  Air  Force  plans  to  procure  80  F-22s  under  lrip,  or  18  percent  of  the 
total  planned  procurement,  at  an  estimated  cost  of  $12.4  biUion,  before 
completing  iot&e.  Although  the  F-22  program  entered  the  engineering  and 
manufacturing  development  phase  before  the  Federal  Acquisition  and 
Streamlining  Act  was  passed,  the  F-22  lrip  quantities  substantially  exceed 
the  10-percent  guideline  included  in  the  act  which  requires  the  Secretary 


■W<»apons  Acquisition:  Low-Rate  Initial  Production  Used  to  Buy  Weapon  Systems  Prematurely 
(GAO/NSIAD-95-18,  Nov.  1994). 
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of  Defense  to  submit  a  specific  explanation  to  the  Congress.  The 
percentage  of  F-22s  to  be  committed  to  production  before  completion  of 
lOT&E  is  higher  than  most  recent  fighter  programs. 

F-22  production  rates  in  the  lrip  phase  of  the  program  are  planned  to 
accelerate  so  that  75  percent  of  the  full-production  rate,  or  36  aircraft  a 
year,  will  be  achieved  under  the  lrip  phase  of  the  program.  We  believe  the 
planned  rate  of  acceleration  exceeds  the  amount  that  is  needed  to 
successfully  complete  the  lrip  phase  of  the  program  and  essentially 
represents  apian  to  commit  to  a  full-rate  production  schedule  before  iot&e 
is  completed.  limiting  lrip  quantities  to  about  six  to  eight  aircraft  a  year, 
or  the  production  rate  that  can  be  supported  by  the  first  set  of  tooling, 
appears  to  be  a  more  prudent  ^proach,  given  the  high  degree  of 
concurrency  now  incorporated  in  the  program  and  the  potential  problems 
associated  with  technological  advances. 

Technology  advances  and  innovations  that  are  critical  to  the  F-22’s 
operational  success  include  an  advanced  architecture  for  the  integrated 
avionics  system,  a  propulsion  system  that  will  allow  cruising  at  supersonic 
speeds  without  the  afterburners  current  fighters  need,  and  low  observable 
(stealth)  technologies  in  an  aircraft  that  is  both  highly  maneuverable  and 
can  travel  at  supersonic  speeds. 

The  need  for  the  F-22,  based  on  our  analysis,  is  not  urgent.  Our  recent 
report  concerning  planned  replacement  of  F-15s  with  F-22s  amply 
demonstrated  that  the  initial  operational  capability  planned  for  the  F-22 
could  be  deferred.^  Moreover,  engine  and  stealthiness  problems  already 
disclosed  by  dod,  and  the  potential  for  avionics  and  software  problems, 
underscore  the  need  to  demonstrate  the  weapon  system’s  performance 
through  flight  testing  before  significant  commitments  are  made  to 
production. 


Concurrency  in  the 
F-22  Program 


In  1990,  DOD  performed  a  statutorily  required  analysis  of  the  concurrency 
in  acquisition  programs  partly  to  define  the  zqipropriate  measures  for 
evaluating  the  degree  of  concurrency  and  associated  risk  in  programs.  The 


^Tactical  Aircraft:  F-15  Replacement  Is  Premature  As  Currently  Planned  (GAO/NSIAD-94-118,  Mar.  26, 
1994). 
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Office  of  the  Secretary  of  Defense  defined  a  highly  concurrent  program®  as 
one  that  proceeds  into  lrip  before  significant  iot&e  is  complete. 

Using  DOD  guidelines,  concurrency  in  the  F-22  program  is  high  because  the 
F-22  program  is  scheduled  to  proceed  into  lrip  well  before  any  iot&e  is 
started.  Further,  considering  the  new  technology  advancements  being 
developed  for  use  in  the  aircraft,  the  level  of  concurrency  increases  the 
cost,  schedule,  and  technical  risks  of  the  program.  We  found  that 
development  flight  tests  of  critical  F-22  technology  advances  are  not 
scheduled  to  begin  until  about  1  year  after  lrip  is  scheduled  to  start  and 
over  $2  billion  will  have  been  committed  to  procure  F-22  aircraft. 

According  to  the  F-22  acquisition  plan,  the  Air  Force  will  commit  to  lrip 
quantities  that  increase  from  4  aircraft  a  year  to  36  a  year  (an  800-percent 
increase),  totaling  80  aircraft,  before  completion  of  lOT&E.  Production  of  36 
aircraft  a  year  under  lrip  represents  75  percent  of  the  planned 
fuU-production  rate.  The  estimated  cost  of  those  80  aircraft  is  $12.4  billion. 
Figure  1  shows  the  planned  schedule  of  commitments  to  procurement  of 
F-22  aircraft  and  the  estimated  cumulative  costs  prior  to  completion  of 
IOT&E. 


’’Definition  in  Report  on  Guidelines  for  Detennining  the  Degree  of  Risk  Appropriate  tor  the 
Development  of  M^or  Defense  Acquisitions  Systems,  and  Assessing  the  Degree  of  Risk  Associated 
with  Various  Degrees  of  Concurrency;  and  Concurrency  in  Mijor  Acquisition  Programs,  April  1990. 
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Figure  1:  Planned  Commitments  to 
Procure  F-22  Aircraft  and  Estimated 
Cumulative  Costs  Prior  to  Completion 
of  lOT&E  (dollars  in  billions) 
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A  first  set  of  hardened  tooling  is  required  initially  to  produce  the 
developmental  aircraft  for  testing.  Program  office  officials  told  us  that  the 
maximum  quantity  of  F-22s  that  can  be  produced  with  the  first  set  of 
tooling  is  about  6  to  8  aircraft  a  year. 
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The  concurrency  of  development,  testing  and  production  in  the  F-22 
program  is  shown  in  figure  2,  which  shows  concurrent  development  and 
production  from  September  1997  through  February  2002. 


Figure  2:  Concurrent  Development  and  Production  of  the  F-22 
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Low-rate  production  of  the  F-22  is  scheduled  to  begin  in  September  1997. 
However,  iot&e  is  not  scheduled  to  take  place  until  December  1999 
through  February  2002.^  Thus,  the  testing  is  not  scheduled  to  be  complete 
until  over  4  years  after  the  start  of  production  and  the  commitment  at  an 
estimated  cost  of  $12.4  billion  to  procure  80  aircraft^  (4  preproduction 


■'Dedicated  lOT&E,  which  is  the  independent  operational  testing  and  evaluation  made  by  an  Air  Force 
test  organization,  is  not  scheduled  to  start  until  March  2001. 

®In  addition  to  the  80  aircraft  planned  under  low-rate  production,  the  Air  Force  plans  to  initiate  long 
lead  effort  on  48  full-rate  production  aircraft  in  September  2001. 
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aircraft  and  76  production  aircraft),  or  18  percent  of  all  442  aircraft  to  be 
procured. 


Flight  Tests  of  Critical 
Technology  Not  Planned 
Until  Well  After  Production 
Start 

Although  laboratory  tests  are  underway  and  simulations  of  the  avionics 
are  planned,  the  Air  Force  does  not  plan  to  flight  test  several  of  the  critical 
F-22  technology  advances  on  an  F-22  until  well  after  the  start  of 
production  in  September  1997.  Flight  tests  of  low  observabihty  are  not 
scheduled  to  begin  until  September  1998.  Although  the  highest  risk 
element  of  the  F-22  program  was  reported  to  be  the  integrated  avionics, 
the  first  flight  test  of  an  F-22  equipped  with  a  complete  integrated  avionics 
system  is  not  scheduled  to  begin  until  September  1999,  2  years  after  the 
start  of  production.  By  the  time  that  testing  begins,  the  Air  Force  will  have 
already  made  commitments  to  procure  20  aircraft  and  long  lead  materials 
for  an  additional  24. 

Plan  for  F-22 

Low-Rate  Production 
Compared  With  1994 
Congressional 
Guidelines 

For  programs  entering  the  engineering  and  manufacturing  phase  of  the 
acquisition  cycle,  the  Federal  Acquisition  Streamlining  Act  of  1994 
requires  the  Secretary  of  Defense  to  explain  to  the  Congress  any  plans  to 
procure  more  than  10  percent  of  the  total  procurement  quantity  in  the  lrip 
phase.  This  provision  of  the  act  is  not  retroactive  to  the  F-22  program. 

In  1991,  when  milestone  n  was  approved  for  the  F-22  program,  the  total 
aircraft  procurement  quantity  planned  was  648.  Accordingly,  10  percent 
would  have  been  65  aircraft.  Currently,  442  aircraft  are  to  be  procured, 
meaning  10  percent  would  be  44  aircraft.  The  number  of  F-22  lrip  aircraft 
currently  planned  is  80,  exceeding  10  percent,  in  either  case. 

Commitments  to 
Production  of  F-22s 
Prior  to  lOT&E 
Completion  Is  Higher 
Than  Most  Prior 
Fighter  Programs 

The  Air  Force’s  planned  commitment  to  production  of  F-22’s  prior  to 
completion  of  iot&e,  as  a  percentage  of  total  production,  exceeds  the 
commitments  made  for  recent  fighter  programs  except  the  F-15,  in  which 
the  percent  is  about  the  same  as  the  F-22.  Figure  3  compares  the  planned 
percentage  of  aircraft  committed  to  production  before  completion  of  iot&e 
for  the  F-22  and  percentages  committed  for  other  recent  fighter  programs.® 

®The  total  aircraft  for  the  F/A-18  includes  aircraft  to  be  procured  through  fiscal  year  1997.  Production 
is  complete  on  all  other  aircraft. 
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Figure  3:  Aircraft  Committed  to 
Production  Before  Compietion  of 
lOT&E  as  a  Percentage  of  Total 
Aircraft 
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Although  the  actual  number  of  F-22  aircraft  to  be  acquired  before 
completion  of  iot&e  is  lower  than  in  the  F-14,  F-15,  F-16,  and  F/A-18,  the 
other  fighters  were  acquired  before  the  end  of  the  Cold  War  when  a 
greater  degree  of  urgency  existed  for  procuring  aircraft. 


Development  Risks 
With  High  Technology 
Systems  and 
Subsystems 


The  Air  Force  plans  to  use  advances  in  technologies  and  innovations  to 
provide  high  performance  and  increased  reliability  and  maintainability  for 
the  F-22.  The  integrated  avionics,  engine,  and  stealth  characteristics  are 
the  primary  areas  that  increase  the  cost,  schedule,  and  technical  risk  in  the 
F-22  program.  After  reviewing  the  program,  the  dod  Defense  Science 
Board  (dsb)  concluded  that  concurrency  was  acceptable  and  risks  were 
readily  controllable,  but  noted  that  the  F-22  program  is  very  ambitious 
technically.  Descriptions  of  some  of  the  problems  that  have  occurred  in 
the  development  program  are  included  below.  The  pxupose  of  these 
descriptions  is  to  illustrate  that  there  remain  important  cost,  schedule,  and 
technical  risks  in  the  F-22  program. 


The  F-22  Program  Office  has  taken  a  number  of  steps  to  reduce  the 
technical  risks  of  the  program,  including  a  54-month 
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Avionics  and  Software  Risk 


demonstration/validation  phase  using  an  F-22  prototype,  and  a  risk 
management  program  for  engineering  and  manufacturing.  Some 
deficiencies  associated  with  the  higher  risk  features  of  the  F-22  have  been 
experienced  during  ground  tests,  requiring  ejqjensive  redesigns. 


The  F-22’s  integrated  avionics  are  expected  to  provide  unprecedented 
situational  awareness  to  the  pilot.  The  F-22  is  the  first  aircraft  to  use 
integrated  avionics,  that  is,  critical  systems  such  as  the  radar,  the  weapons 
management  system,  and  electronic  warfare  sensors  that  work  as  one  umt. 
The  key  to  achieving  the  necessary  performance  is  the  successful 
development  of  highly  advanced  integrated  computer  processors,  known 
as  the  common  integrated  processors,  and  large  amounts  of  software. 

Avionics  and  software  integration  has  been  characterized  by  the  dod 
Defense  Acquisition  Board  as  one  of  the  highest  risks  to  the  successful 
development  of  the  F-22.  The  risk  assessment  was  prepared  for  the  dod 
Defense  Acquisition  Board  to  evaluate  the  readiness  of  the  F-22  to  begin 
the  engineering  and  manufacturing  development  phase  of  the  acquisition 
cycle  in  1991.  This  report  in  June  1991  explained  that  the  estimated 
1.3  million  single  lines  of  software  code  needed  for  the  F-22  represented 
the  largest  sofbware  task  ever  for  an  attack/fighter  onboard  software 
program.  Further,  the  dsb  in  1993  rated  the  integrated  avionics  as  the 
highest  technical  risk  in  the  F-22  program.  Program  managers  for  the  F-22 
agreed  in  October  1994  that  the  avionics  and  software  integration  are  the 
most  risky  tasks  facing  the  contractors. 

In  a  separate  report,  we  concluded  in  1994  that  although  the  Air  Force’s 
planned  strategy  for  the  F-22  software  was  generally  sound  in  concept, 
some  significant  features  of  the  strategy  were  not  being  followed.^  For 
example,  the  independent  verification  and  validation  of  software 
products — ^part  of  the  quality  assurance  process — ^was  less  rigorous  than 
planned.  In  addition,  the  technical  risks  being  encountered  with  the 
system/software  engineering  environment  and  common  integrated 
processor  were  not  being  formally  reported  to  dod  management.  Finally, 
we  indicated  that  the  Air  Force  had  begun  actions  to  respond  to  our 
concerns. 

DOD  responded  to  that  report  in  February  1996,  indicating  that  the  quality 
assurance  program  is  now  being  complied  with  as  planned,  dod  also  stated 
that  common  automated  tools  had  matured  and  would  support  completion 


^Air  Force  F-22  Embedded  Computers  (GAO/AIMD-94-177R,  Sept.  23, 1994). 
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of  the  software  development  effort  through  the  engineeilng  and 
manufacturing  development  phase  of  the  program.  We  have  not  verified 
the  DOD  response. 


Engine  Problems  The  F-22’s  engine  has  not  been  flight  tested,  but  has  experienced  problems 

during  ground  tests.  The  F-22’s  engine  is  expected  to  be  the  first  to  provide 
the  abiUty  to  fly  faster  than  the  speed  of  sound  for  an  extended  period  of 
time  without  the  high  fuel  consumption  characteristic  of  aircraft  that  use 
afterburners  to  achieve  supersonic  speeds.  It  is  expected  to  provide  high 
performance  and  high  fuel  efficiency  at  slower  speeds  as  well. 

Problems  with  performance  of  the  F-22’s  engine  first  surfaced  after  the 
initial  engine  ground  tests  began  in  December  1992.  The  contractor  is 
conducting  a  series  of  interim  tests,  with  a  goal  of  having  a  complete 
engine  with  a  redesigned  turbine  and  other  changes  qualified  for  flight  by 
December  1996  if  tests  now  planned  for  1995  are  successfiil.  If  not,  F-22 
flight  tests  will  be  started  with  an  engine  that  is  not  fully  representative  of 
the  current  approved  configuration. 

An  Executive  Independent  Review  Team  was  formed  to  provide  advice  on 
engine  development  issues,  including  a  turbine  problem.  The  team  stated 
that  it  did  not  consider  the  nature  and  number  of  engine  problems  to  be 
excessive  for  a  highly  sophisticated  engine  at  this  stage  of  development. 
They  also  stated  that  the  proposed  solutions  can  only  be  proven  by 
exposing  authentic  hardware  to  the  full  range  of  realistic  testing. 

Through  November  1994,  the  Air  Force  had  identified  engine  problems 
that  may  cost  as  much  as  $479  million  to  remedy.  The  Air  Force  increased 
the  target  cost  of  the  engine  development  contract  by  $218  million  to 
design  and  test  solutions  to  the  engine  problems.  The  incorporation  of 
corrective  modifications  to  future  production  engines  is  e3q)ected  to 
increase  production  costs  by  $123  million.  The  Air  Force  believes  its 
current  program  estimate  can  cover  the  $341  million  increase 
($218  million  plus  $123  million),  but  the  Air  Force  has  identified  other 
potential  design  changes  that  may  add  $138  miUion  to  development  and 
production  costs.  The  other  potential  design  changes  are  not  currently 
part  of  the  planned  program. 


Problems  \\^th  Stealthiness  The  low  observability  or  stealth  characteristics  of  the  F-22  is  another  risk 

area  The  F-22  is  to  be  the  first  supersonic,  highly  maneuverable  fighter 
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that  uses  low  observable  technologies  to  reduce  radar,  infrared,  acoustic 
and  optical  signatures  of  the  aircraft,  making  it  difficult  for  an  adversary  to 
detect. 

An  evaluation  of  the  complete  F-22  radar  signature  using  computer  models 
and  a  scale  version  of  the  aircraft  concluded  that  the  aircraft’s  radar 
signature  did  not  meet  the  Air  Force’s  operational  requirement.  Although 
DOD  advised  us  that  these  problems  were  not  considered  major,  design 
changes,  such  as  reducing  the  number  of  aircraft  maintenance  access 
panels  and  fuel  drain  holes,  and  reshaping  the  airfirame  were  evaluated 
through  December  1994  to  determine  if  these  changes  were  successful  in 
reducing  the  signature,  dod  further  stated  that  the  contractual 
specifications  are  being  revised.  'The  estimated  development  cost  to 
resolve  these  problems  is  about  $20  million  according  to  the  F-22  program 
office.  Additional  production  costs  of  about  $110  million  could  also  be 
required,  however,  program  officials  told  us  the  total  estimated  cost 
($71.6  billion)  of  the  F-22  program  should  not  be  affected. 


Other  Technical  Risks 


The  DSB,  in  its  review  of  the  F-22  program’s  concurrency  and  technical 
risks,  identified  a  number  of  other  concerns.  Examples  of  concerns 
mentioned  by  the  dsb  include 

•  control  of  excess  aircraft  weight; 

•  use  of  new  materials  and  fabrication  processes; 

•  uncertain  durability  of  composite  materials  in  the  F-22  application; 

•  probable  inability  of  the  engine  to  meet  performance  and  durability  goals 
before  first  flight; 

•  design  of  certain  low  observable  features  and  applicable  manufacturing 
processes; 

•  very  challenging  development  of  electronic  warfare  system; 

•  late  scheduling  of  tests  relative  to  increasing  production  to  12  aircraft  a 
year;  and 

•  the  need  for  a  long,  evolutionary  software  development. 

Overall,  the  dsb  characterized  the  F-22  program  as  very  ambitious 
technically. 


Recommendation 


We  recommend  that  the  Secretary  of  Defense  reduce  the  degree  of 
concurrency  in  the  program  because 
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•  independent  testing  of  technology  advances  (iot&e),  will  not  be  completed 
before  significant  commitments  are  made  to  produce  F-22s; 

•  the  percentage  of  planned  F-22s  to  be  committed  to  production  before 
completion  of  iot&e  is  higher  than  most  recent  fighter  programs;  and 

•  the  need  for  the  F-22  is  not  urgent. 

To  minimize  commitments  to  production  of  F-22s  until  after  successful 
completion  of  iot&e,  we  recommend  that  the  Secretary  of  Defense  limit 
LRiP  quantities  to  that  which  can  be  produced  using  the  first  set  of  hard 
tooling,  about  six  to  eight  aircraft  a  year. 


DOD  partially  agreed  with  the  findings  in  this  report,  but  disagreed  with  the 
recommendations,  dod  indicated  it  believed  that  the  F-22  program  had  an 
acceptable  degree  of  concurrency  based  on  the  dsb’s  evaluation  that  risks 
associated  with  premature  entry  into  successively  higher  rates  of 
production  were  readily  controllable  through  insistence  on  meeting 
certain  key  events  and  test  criteria  already  built  into  the  F-22  plan. 

The  record  shows,  however,  that  dod  has  often  been  unwilling  or  unable  to 
ciutail  production  of  other  systems  after  it  starts,  despite  discovery  of 
significant  problems  in  development  or  operational  tests.  We  believe  the 
degree  of  concurrency  in  the  program  should  be  addressed  now,  because 
(1)  independent  testing  of  important  technology  advances  is  not  planned 
until  after  commitments  are  made  to  produce  F-22s,  (2)  program 
concurrency  is  high  according  to  dod’s  prescribed  measure,  and  (3)  the 
need  for  the  F-22  is  not  urgent. 

dod  disagreed  with  (1)  our  use  of  the  completion  of  iot&e  as  a  measure  of 
concurrency  and  risk  in  the  program,  (2)  our  positions  on  the  level  of  risk 
in  the  F-22  program,  and  (3)  the  comparison  of  the  F-22  to  prior  fighter 
programs  using  the  percentage  of  planned  aircraft  procured  during  lrip  as 
a  measurement. 

We  first  jqiplied  dod’s  own  guidance  for  measuring  the  degree  of 
concurrency  in  the  program,  that  is,  the  amoimt  of  iot&e  completed  prior 
to  entering  productions.  We  also  used  other  metrics,  such  as  the  percent  of 
the  total  program  committed  to  production  before  completion  of  iot&e. 
Further,  dod’s  comments  appear  to  discount  the  risks  in  the  program 
identified  by  the  dsb.  Our  comparison  of  the  F-22  to  recent  fighter 
programs,  although  not  the  same  as  comparisons  made  by  the  dsb. 


Agency  Comments 
and  Our  Evaluation 
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provides  an  important  historical  perspective  that  dod’s  planned  lrip  meets 
or  exceeds  the  fighter  programs  undertaken  during  the  Cold  War. 


Scope  and 
Methodology 


In  conducting  our  work,  we  obtained  information  and  interviewed  officials 
from  the  Office  of  the  Secretary  of  Defense,  DSB,  and  Air  Force 
Headquarters,  Washington,  D.  C. ;  F-22  System  Program  Office, 
Wright-Patterson  Air  Force  Base,  Ohio;  the  Air  Force  Air  Combat 
Command,  Langley  Air  Force  Base,  Virgiiua;  and  the  Air  Force  Operational 
Test  and  Evaluation  Center,  Kirtland  Air  Force  Base,  New  Mexico. 

We  interviewed  officials  in  charge  of  program  management,  the  operation 
of  tactical  fighter  aircraft,  risk  assessment  and  the  operational  testing  of 
Air  Force  weapon  systems.  We  reviewed  documents,  including  program 
office  briefings,  program  schedules,  test  plans  and  reports,  technology  risk 
assessments,  requirements  documents  and  cost  reports.  We  used  these 
interviews  and  documents  to  determine  the  program  management 
philosophy,  the  amount  of  program  concurrency,  the  planned  flight  testing 
of  F-22  technologies,  program  technology  requirements  and  program  risk 
assessments. 

We  also  reviewed  dod  instructions.  Air  Force  regulations.  Office  of 
Secretary  of  Defense  guidance,  publications  firom  the  Defense  Systems 
Management  College,  our  prior  reports,  a  report  of  another  audit 
organization,  congressional  reports,  an  Institute  for  Defense  Analyses 
report,  a  dsb  report,  a  report  prepared  for  the  Defense  Acquisition  Board, 
executive  summaries,  and  monthly  program  reports. 

In  addition,  we  interviewed  officials  firom  the  Air  Force’s  Air  Combat 
Command  and  examined  the  F-22  System  Operational  Requirements 
Document,  Statements  of  Need,  and  the  Mission  Element  Need  Statement 
for  new  fighter  aircraft. 

We  performed  our  work  from  August  1994  through  February  1995  in 
accordance  with  generally  accepted  government  auditing  standards. 


We  are  sending  copies  of  this  report  to  the  Secretaries  of  Defense  and  the 
Air  Force  and  the  Director  of  the  Office  of  Management  and  Budget. 
Copies  will  also  be  made  available  to  others  on  request. 
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B-259204 


Please  contact  me  on  (202)  512-4841  if  you  or  your  staff  have  any 
questions  concerning  this  report.  Major  contributors  to  this  report  are 
listed  in  appendix  II. 


Louis  J.  Rodrigues 
Director,  Systems  Development 
and  Production  Issues 
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List  of  Congressional  Committees 

The  Honorable  Strom  Thurmond 
Chairman 

The  Honorable  Sam  Nunn 
Ranking  Minority  Member 
Committee  on  Armed  Services 
United  States  Senate 

The  Honorable  Ted  Stevens 
Chairman 

The  Honorable  Daniel  K  Inouye 
Ranking  Minority  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
United  States  Senate 

The  Honorable  Floyd  Spence 
Chairman 

The  Honorable  Ronald  V.  DeUums 
Ranking  Minority  Member 
Committee  on  National  Security 
House  of  Representatives 

The  Honorable  C.  W.  Bill  Young 
Chairman 

The  Honorable  John  P.  Mmtha 
Ranking  Minority  Member 
Subcommittee  on  National  Security 
Committee  on  Appropriations 
House  of  Representatives 
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See  comment  1 . 


OFFtCE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 

3000  DEFENSE  PENTAGON 
WASHINGTON  DC  20301-3000 

FEB  2  4  1995 

Mr.  Louis  J.  Rodrigues 
Director,  Systems  Development  and 
Production  Issues 

National  Security  and  International 
Affairs  Division 
U.S.  General  Accounting  Office 
Washington,  D.C.  20548 


AcaunmoN  and 
TCCHNOLOGY 


Dear  Mr.  Rodrigues: 

This  is  the  Department  of  Defense  (DoD)  response  to  the 
General  Accounting  Office  (GAO)  draft  report,  entitled 
"TACTICAL  AIRCRAFT:  Concurrency  in  Development  and 
Production  of  F-22  Aircraft  Should  be  Reduced, "  dated 
January  18,  1995  (GAO  Code  707023/OSD  Case  9847) .  The  DoD 
partially  concurs  with  the  report  findings,  but  nonconcurs 
with  the  recommendations. 


Language  contained  in  the  Senate  Armed  Services 
Committee  report  on  the  National  Defense  Authorization  Act 
for  FY  1995,  Report  102-212,  requested  the  Secretary  of 
Defense  and  the  GAO  to  independently  assess  the  concurrency 
and  risk  in  the  F-22  program.  The  Department  convened  a 
Defense  Science  Board  (DSB)Task  Force  to  perform  an 
independent  assessment  as  a  part  of  the  DoD  overall 
evaluation.  The  DSB  conducted  an  assessment  of  a  number  of 
concurrency  metrics  and  compared  them  to  the  current  DoD 
guidelines .  The  DSB  also  reviewed  the  current  technical 
status  of  the  program  in  order  to  determine  the  risk,  based 
upon  the  progress  to  date  and  the  contractual  objectives. 

Although  both  the  DSB  and  the  GAO  should  have  had 
essentially  the  same  data  base  to  evaluate,  the  conclusions 
reached  by  the  DSB  differ  from  those  of  the  GAO.  There 
appear  to  be  three  main  reasons  for  this  difference: 

(1)  the  degree  of  relevance  the  GAO  attached  to  the 
completion  of  dedicated  Initial  Operational  Test  and 
Evaluation  (lOTSE)  as  a  measure  of  concurrency  and/or  risk 
in  a  program;  (2)  the  GAO  position  on  the  level  of  risk  in 
the  F-22  program;  and  (3)  the  very  narrow  metric  the  GAO 
chose  in  its  comparison  of  F-22  concurrency  with  prior 
fighter  development  programs. 

One  of  the  key  points  of  the  April  17,  1990,  DoD  report 
prepared  in  compliance  with  Section  801  of  the  National 
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See  comment  1 . 


Defense  Authorization  Act  for  Fiscal  Years  1990  and  1991  is 
the  importance  of  continually  assessing  the  risks  associated 
with  the  level  of  concurrency  planned  for  a  program  to 
assure  a  proper  balance  with  other  program  parameters.  The 
report  even  provides  the  guidelines  for  achieving  a  proper 
balance  between  concurrency  and  risk. 

While  the  GAO  stated  that  the  concurrency  guidelines 
outlined  in  the  April  1990  report  were  used  in  its 
assessment  to  establish  the  degree  of  concurrency  in  the 
F-22  program,  the  GAO  did  not  acknowledge  that  the  F-22 
program  meets  most,  if  not  all,  of  the  risk/concurrency 
guidelines  established  in  the  report  for  maintaining  the 
proper  balance  between  concurrency  and  risk  in  development. 
The  GAO  focused  almost  exclusively  on  the  number  of  aircraft 
committed  to  production  prior  to  completion  of  dedicated 
IOT4E  as  its  sole  metric  for  judging  concurrency  and  risk  in 
the  program. 

In  assessing  concurrency  and  risk  for  the  F-22,  the 
focus  is  more  properly  directed  at  the  efficiencies  and 
economies  associated  with  the  production  ramp-up  rate 
decision.  Under  the  planned  F-22  development  plan,  the  DSB 
concluded  that  the  F-22  will  have  passed  the  critical  tests 
on  all  the  major  systems  and  subsystems  prior  to  the  ramp-up 
decision.  If  the  tests  are  not  passed,  the  DSB  observed 
that  production  can  be  constrained. 

The  detailed  DoD  comments  on  the  report  findings  and 
recommendations  are  provided  in  the  enclosure.  The  DoD 
appreciates  the  opportunity  to  comment  on  the  draft  report. 

Sincerely, 

George  R.  Schneiter 

Director 

Strategic  &  Tactical  Systems 


Enclosure 
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GAO  DRAFT  REPORT  -  DATED  JANUARY  18,  1995 
GAO  Code  707023/OSD  CASE  9847 

"TACTICAL  AIRCRAFT:  CONCURRENCY  IN  DEVELOPMENT  AND 
PRODUCTION  OF  F-22  AIRCRAFT  SHOULD  BE  REDUCED" 

DEPARTMENT  OF  DEFENSE  COtMENTS 

***** 

FINDINGS 


FINDING  A:  Statue  of  the  F-22  Low  Rate  Initial  Production 
Proaram.  The  GAO  used  a  statutorily  required  DoD  guide, 
dated  T^ril  1990,  to  measure  the  degree  of  concurrency  in 
the  F-22  program.  The  GAO  indicated  that  the  measure  of 
concurrency  is  the  amount  of  initial  operational  test  and 
evaluation  (lOTSE)  completed  before  a  system  enters 
production.  The  GAO  explained  that  initial  operational 
tests  are  intended  to  demonstrate  the  system's  effectiveness 
and  suitability  for  military  use.  The  GAO  observed  that  the 
purchase  of  long  lead  materials  to  support  the  initiation  of 
F-22  low  rate  initial  production  (LRIP)  is  planned  for 
August  1996,  and  the  LRIP  contract  award  is  scheduled  for 
September  1997.  The  GAO  explained  that  LRIP  aircraft  are 
those  aircraft  procured  during  the  period  of  concurrency. 

The  GAO  observed  that,  in  the  FY  1990-1991  National  Defense 
Authorization  Act,  the  conferees  indicated  that  LRIP 
quantities  should  not  total  a  significant  percentage  of  the 
total  planned  procurement  for  a  particular  weapon  system. 

The  GAO  also  observed  that  the  Federal  Acquisition 
Streamlining  Act  of  1994  prescribed  new  controls  for  LRIP, 
and  requires  the  Secretary  of  Defense  to  explain  to  the 
Congress  why  any  planned  LRIP  quantities  exceed  10  percent 
of  the  planned  production  quantity  as  defined  at  the 
Milestone  II  or  development  decision.  The  GAO  found  that 
the  DoD  plans  to  procure  442  F-22  aircraft  and  that  the 
number  of  LRIP  aircraft  currently  planned  is  80,  which 
exceeds  the  10  percent  (or  44  aircraft)  specified  in  the 
Act.  However,  the  GAO  acJcnowledged  that  the  Act  was  not  in 
effect  when  the  F-22  program  reached  Milestone  II  and  is  not 
retroactive  to  the  program. 

The  GAO  also  found  that  the  DoD  planned  production 
commitment  of  the  F-22  prior  to  completion  of  lOT&E,  as  a 
percentage  of  total  production,  exceeds  the  commitments  made 
for  prior  fighter  programs,  except  the  F-15,  which  was  about 
the  same  as  the  F-22.  However,  the  GAO  asserted  that  the 

Enclosure 
page  1  of  10 
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Nowon  pp.  1-3  and  7-8. 


See  comment  3. 


See  comment  4. 


prior  fighter  programs,  including  the  F-14,  F-15,  F-16,  and 
F/A-18,  were  acquired  during  the  Cold  War  when  a  greater 
degree  of  urgency  existed  for  procuring  aircraft.  (pp.  1-3, 
pp.  12-14/GAO  Draft  Report) 

DOD  BESPCXiSE:  Partially  concur.  The  DoD  agrees  that  the 
currently  planned  LRIP  quantities  for  the  F-22  exceed  the  10 
percent  guideline  called  for  in  the  Federal  Acquisition 
Streamlining  Act  of  1994.  However,  as  the  GAO  has  pointed 
out,  the  F-22  Milestone  II  decision  was  completed  prior  to 
the  enactment  of  that  guideline.  In  the  case  of  the  F-22, 
rather  than  focus  on  an  arbitrary  guideline  that  varies  with 
production  quantities,  and  does  not  even  apply,  the  most 
important  consideration  should  be  whether  the  proposed  ramp- 
up  rate  for  the  aircraft  ma)ces  sense  and  is  economically 
sound.  The  Defense  Science  Board  (DSB)  concluded  that  the 
F-22  will  have  passed  its  critical  tests  prior  to  the  ramp- 
up,  and,  if  it  has  not,  the  production  can  be  constrained 
until  the  tests  have  been  passed. 

The  data  base  on  prior  fighter  aircraft  programs  shows  that 
the  number  of  aircraft  procured  prior  to  Milestone  III 
(full-rate  production)  by  the  F-16,  F/A-IB,  and  F-15 
programs  is  in  line  with  the  planned  F-22  procurement.  A 
comparison  of  the  most  recent  fighter  aircraft  development 
programs  shows  that  the  DoD  committed  to  production  of  92  F- 
ISs,  107  F-16S,  and  113  F/A-lSs  by  Milestone  IIIB.  The  F-22 
program  will  commit  to  80  aircraft  prior  to  the  same 
milestone. 

The  important  issue  is  the  balance  In  the  development  and 
production  programs.  In  that  respect,  the  F-22  program 
employs  most,  if  not  all,  of  the  rislc/concurrency  guidelines 
called  for  in  the  April  17,  1990,  report  for  maintaining  a 
proper  balance  between  concurrency  and  ris]c  in  a  program. 
Unli)«e  prior  fighter  developments,  the  program  is  an  event- 
driven  program  that  explicitly  lin)cs  milestone  decisions  to 
demonstrated  accomplishments.  As  the  DSB  discovered  during 
its  assessment  of  the  F-22  program,  appropriate  testing  is 
accomplished  prior  to  any  significant  commitment  of 
production  resources. 


See  comment  3. 


FINDING  B;  Concurrency  in  the  F-22  Program  Is  High. 

Using  DoD  guidelines,  the  GAO  concluded  that  concurrency  in 
the  F-22  program  is  high,  because  the  F-22  program  is 
scheduled  to  proceed  into  LRIP  well  before  any  lOTSE  is 
started.  Further,  considering  the  new  technology  advance¬ 
ments  being  developed  for  use  in  the  aircraft,  the  GAO 
concluded  that  the  level  of  concurrency  increases  the  cost. 


Enclosure 
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schedule/  and  technical  risks  of  the  program.  The  GAO  found 
that  development  flight  tests  of  critical  F-22  technology 
advances  are  not  scheduled  to  begin  until  about  one  year 
after  LRIP  is  scheduled  to  start  and  substantial  commitments 
are  made  to  procure  F-22  aircraft.  For  example,  the  GAO 
found  that  the  Air  Force  will  have  committed  to  procure  8 
aircraft  and  long  lead  materials  for  an  additional  12  air¬ 
craft  before  testing  the  basic  avionics.  The  GAO  also  found 
that  the  first  flight  test  of  an  F-22  equipped  with  a 
complete  integrated  avionics  system — the  highest  risk 
element  of  the  F-22  program — is  not  scheduled  until  more 
than  3  years  after  production  activities  start.  The  GAO 
concluded  that,  by  that  time,  the  Air  Force  will  have 
already  made  commitments  to  procure  20  aircraft  and  long 
lead  materials  for  an  additional  24  aircraft. 

The  GAO  also  observed  that  LRIP  of  the  F-22  is  scheduled  to 
begin  in  September  1997.  However,  the  GAO  noted  that  lOTSE 
is  not  scheduled  to  take  place  until  December  1999  through 
February  2002.  The  GAO  found  that,  according  to  the  F-22 
acquisition  plan,  the  Air  Force  will  commit  to  LRIP 
quantities  that  increase  from  4  aircraft  per  year  to  36  per 
year  (a  800  percent  increase) ,  totaling  80  aircraft  at  a 
estimated  cost  of  $12.4  billion,  before  completion  of  lOTSE. 
The  GAO  concluded  that  production  of  36  aircraft  per  year 
under  LRIP  represents  75  percent  of  the  planned  full 
production  rate.  The  GAO  further  concluded  that  the  F-22 
operational  test  schedule  is  still  tentative,  because  the 
Director,  Operational  Test  and  Evaluation,  Office  of  the 
Secretary  of  Defense,  disapproved  the  F-22  Test  and 
Evaluation  Master  Plan  (TEMP)  on  March  31,  1994.  According 
to  the  GAO,  the  Director  stated  that  the  plan  was  not 
adequate  to  determine  the  F-22's  operational  effectiveness 
and  suitability  for  combat  and  called  for  the  Air  Force  to 
submit  a  revised  test  plan  in  February  1995.  (pp.  6-12/GAO 
Draft  Report) 

POD  RESPONSE;  Nonconcur.  With  regard  to  concurrency,  the 
i^ril  17,  1990,  DoD  report  to  the  Congress  presented 
guidelines  for  determining  the  degree  of  concurrency  that  is 
appropriate  for  major  defense  acquisition  programs.  The 
report  states  at  the  outset  that. .. "concurrency,  per  se,  is 
not  necessarily  an  undesirable  program  feature.  Total 
avoidance  of  concurrency  would  lead  to  a  series  of  time 
consuming  and  costly  gaps  in  a  program  as  it  evolves  from 
development  to  full-scale  production.  The  appropriate 
amount  of  concurrency  to  build  into  a  program  is  a  matter  of 
judgment  involving  trade-offs  cunong  risk,  cost  and  schedule 
to  achieve  a  specified  level  of  performance."  Recognition 
of  the  appropriateness  of  concurrency  to  major  acquisition 
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programs  is  an  essential  element  of  the  assessment  which  the 
GAO  apparently  ignored. 

The  DSB  concluded  that  the  concurrency  in  the  F-22  program 
was  conservative  compared  with  other  tactical  fighters.  The 
DSB  conclusions  were  based  on  examination  of  a  very  broad 
data  base,  which  included  data  originally  prepared  by  the 
Institute  for  Defense  Analyses  (IDA)  in  support  of  the 
independent  risk  assessment  of  the  F-22  by  the  DoD  prior  to 
the  Milestone  II  decision  in  June  1991  (and  updated  in  1994 
based  on  the  program  re-phasing) ;  F-22  data  from  Lockheed 
Premise  Modification  to  Master  Schedule,  No.  17,  dated 
November  16,  1994;  RAND  project  memorandum  "Use  of  Flight 
Test  Results  in  Support  of  F-22  Production  Decision,"  dated 
November  1994;  and  other  concurrency  metrics.  The  metrics 
examined  by  the  DSB  included:  (1)  Cumulative  Production 
Commitments  (number  of  aircraft)  vs.  Proportion  of 
Development  Test  and  Evaluation  (DTiE)  Flight  Testing 
Completed;  (2)  Cumulative  Production  Commitments  (value  of 
aircraft  in  millions  of  dollars)  vs.  Proportion  of  DTSE 
Flight  Test  Completed;  (3)  Key  Periods  of  Development  for 
Fighter/Attack  Aircraft;  (4)  Flight  Test  Data  Comparisons; 

(5)  Major  Problem  Occurrence  vs.  Percentage  of  Flight  Test 
Accomplished;  (6)  Production  Start  vs.  Completion  of  lOTSE; 
and  (7)  Aircraft  Procured  Prior  to  Completion  of  lOTSE.  The 
GAO  report  focused  on  a  very  narrow  data  base  that  did  not 
adequately  consider  a  range  of  potential  concurrency 
metrics. 

It  is  not  appropriate  to  focus  on  completion  of  dedicated 
I0T4E  as  the  key  element  in  determining  whether  production 
ramp-up  for  the  F-22  is  properly  timed.  There  are  two 
principal  reasons.  First,  the  purpose  of  OTSE  is  not  to 
uncover  major  developmental  problems.  The  RAND  study  showed 
that  critical  problems  are  generally  uncovered  in  the  first 
10-20  percent  of  developmental  testing.  The  purpose  of  OTsE 
is  to  confirm  that  line  pilots  and  maintenance  personnel  can 
use  the  system  to  achieve  the  operational  objective  for 
which  it  is  designed.  Second,  the  OTSE  community  (the 
Director  of  Operational  Test  and  Evaluation  (DOTSE) ,  and  the 
Air  Force  Operational  Test  and  Evaluation  Center)  will  be 
involved  throughout  the  acquisition  process,  not  just  during 
dedicated  lOTSE.  The  DOTSE  is  already  engaged  in  monitoring 
F-22  progress  and  will  provide  on-going  operational 
assessments  through  its  Annual  Report  to  Congress. 
Additionally,  the  Air  Force  will  generate,  and  the  DOTSE 
will  approve,  a  formal  Early  Operational  Assessment  in 
support  of  the  1998  LRIP  decision.  All  testing,  to  include 
ground  testing  of  subsystems  and  flight  testing  of  delivered 
aircraft,  will  be  considered  in  that  assessment.  Note  that 
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seven  of  nine  F-22  development  aircraft  will  be  flying  by 
the  time  the  report  is  completed,  and  significant  avionics 
testing  will  have  been  completed  in  both  the  Flying  Test  Bed 
and  the  F-22s. 

The  DSB  identified  the  key  events  for  each  critical 
technology  advance/subsystem,  and  examined  the  current 
status  and  planned  milestones  for  each.  The  data  showed 
that  for  each  technical  area,  several  key  test  events  will 
be  completed  prior  to  Lot  2  long-lead  and  contract  award. 
That  is  consistent  with  the  event-driven  strategy  for  the 
program.  The  DSB  concluded  that  those  test  milestones  were 
appropriate  for  judging  readiness  for  production  rarop-up. 

The  DSB  further  stated  that  given  the  amount  and  duration  of 
testing  (in  the  F-22  program),  it  seemed  unlikely  that 
keying  the  start  of  production  ramp-up  to  completion  of 
dedicated  lOTtE  testing  would  substantially  reduce  technical 
risk. 

The  GAO  implication  that  disapproval  of  the  F-22  TEMP  was 
related  to  concurrency  Issues  is  misleading.  The  fact  that 
the  TEMP  was  returned  without  approval  is  irrelevant  to 
concurrency  or  risk  discussions.  The  lOT&E  schedule  is  not 
the  issue  in  the  DOTSE  review  of  the  TEMP.  Rather,  the 
content  of  the  lOTSE  program,  including  the  Under  Secretary 
of  Defense  (Acquisition  and  Technology)  directed  tests  and 
the  proper  description  of  those  efforts,  is  the  issue  that 
has  delayed  approval  of  the  TEMP. 


FIND IMG  C;  Development  Risks  with  High  Technology  Bystems 
end  Subeyeteme.  The  GAO  concluded  that  integrated  avionics, 
engine,  wd  stealth  characteristics  are  the  primary  areas 
that  Increase  the  cost,  schedule,  and  technical  risk  in  the 
F-22  program.  However,  the  GAO  noted  that  the  F-22  Program 
Office  has  taken  a  number  of  steps  to  reduce  the  technical 
risks  of  the  program,  including  a  54-month  Demonstration/ 
Validation  phase  using  an  F-22  prototype,  and  a  risk 
management  program  for  engineering  and  manufacturing. 
Nevertheless,  the  GAO  found  that  some  deficiencies 
associated  with  the  higher-risk  features  of  the  F-22  have 
been  experienced  during  ground  tests,  requiring  expensive 
redesigns. 

The  GAO  observed  that  the  F-22  is  the  first  aircraft  to  use 
integrated  avionics — i.e.,  critical  systems,  such  as  the 
radar,  the  weapons  management  system,  and  electronic  warfare 
sensors  that  work  as  one  unit.  The  GAO  asserted  that  the 
key  to  achieving  the  necessary  performance  is  the  successful 
development  of  highly  advanced  integrated  computer 
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processors,  known  as  common  integrated  processors,  and  large 
amounts  of  software.  The  GAO  reported  in  1994  (OSD  Case 
9494-A)  that,  although  the  planned  strategy  for  the  F-22 
software  was  generally  sound  in  concept,  some  significant 
features  of  the  strategy  were  not  being  followed — e.g.,  the 
independent  verification  and  validation  of  software 
products,  which  is  part  of  the  quality  assurance  process, 
was  less  rigorous  than  initially  planned.  The  GAO  also 
reported  that  common  automated  tools  for  software 
development  were  not  working  as  intended  in  the  F-22 
program. 

The  GAO  also  observed  that,  although  the  F-22  engine  has  not 
been  flight  tested,  problems  have  been  experienced  during 
ground  tests.  The  GAO  indicated  that  the  F-22  engine  is 
expected  to  be  the  first  to  provide  the  ability  to  fly 
faster  than  the  speed  of  sound  for  an  extended  period  of 
time  without  the  high  fuel  consumption  characteristic  of 
aircraft  that  use  afterburners  to  achieve  supersonic  speeds. 
The  GAO  also  indicated  that  the  F-22  engine  is  expected  to 
provide  high  performance  and  high  fuel  efficiency  at  slower 
speeds  as  well.  The  GAO  found  that  problems  with  engine 
performance  first  surfaced  after  the  initial  engine  ground 
tests  began  in  December  1992.  The  GAO  observed  that  the 
contractor  is  conducting  a  series  of  interim  tests  with  the 
goal  of  having  a  complete  engine  with  the  redesigned  turbine 
and  other  changes  qualified  for  flight  by  December  1996  if 
tests  currently  planned  for  1995  are  successful.  If  not, 
the  GAO  concluded  that  the  F-22  flight  tests  will  be  started 
with  an  engine  that  is  not  fully  representative  of  the 
current  approved  configuration.  The  GAO  also  noted  that  the 
identified  engine  problems  may  cost  as  much  as  $479  million 
to  remedy. 

Finally,  the  GAO  observed  that  low  observability  or  stealth 
characteristics  of  the  F-22  is  another  risk  area.  The  GAO 
explained  that  the  F-22  is  to  be  the  first  supersonic, 
highly  maneuverable  fighter  that  uses  low  observable 
technologies  to  reduce  radar,  infrared,  acoustic  and  optical 
signatures  of  the  aircraft,  making  it  difficult  for  an 
adversary  to  detect.  The  GAO  found  that  an  evaluation  of 
the  complete  F-22  radar  signature  concluded  the  signature 
did  not  meet  the  contract  specification.  The  GAO  also  found 
that  the  estimated  development  cost  to  resolve  the  signature 
problem  is  $20  to  $25  million  unless  additional  changes  are 
reqpjired.  (pp.  16-20/GAO  Draft  Report) 


DOD  RESPONSE:  Partially  concur.  The  DoD  agrees  that  the 
primary  areas  of  risk  in  the  F-22  program  are  integrated 
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avionics,  engine,  and  stealth.  Those  areas  were  identified 
by  the  Office  of  the  Secretary  of  Defense  (OSD)  in  its 
independent  risk  assessment  conducted  prior  to  the  Milestone 
II  decision  in  1991.  The  DoD,  however,  does  not  concur  with 
any  of  the  other  GAO  assertions  regarding  the  current  level 
of  risk  in  the  program. 

The  DSB  examined  the  key  event  milestones  for  each  technical 
area  of  the  F-22  program  with  particular  emphasis  on  the 
critical  technology  advances  and  subsystems.  The  DoD 
acknowledges  that  the  engine  is  currently  deficient  in 
performance,  particularly  subsonic  specific  fuel 
consun¥>tion,  which  has  resulted  in  the  redesign  of  the  high- 
pressure  turbine  and  exhaust  nozzle.  Also,  the  vibratory 
stress  levels  in  the  fan  and  turbine  are  higher  than 
expected.  The  DSB  found  that  fixes  for  all  those 
deficiencies  are  currently  being  implemented  and  should 
rectify  the  current  problems.  The  DSB  found  that  the  key 
event  for  the  engine  is  a  test  of  the  redesigned  engine 
scheduled  for  inld-1995.  That  is  a  key  decision  point  with 
regard  to  technical  and  schedule  risk. 

Contrary  to  the  GAO  assertion,  there  have  been  no  major 
problems  in  the  observability  areas  of  the  F-22.  The 
Program  Office  conducted  a  comprehensive  review  of  the 
contractual  specifications  in  the  observability  area,  and 
concluded  that  there  was  some  over-specification  that  could 
be  eliminated.  As  a  result,  the  contractual  specification 
is  being  revised,  but  without  jeopardizing  the  fundamental 
low  observability  requirements  for  the  aircraft.  The  DSB 
review  concluded  that  there  was  no  unacceptable  concurrency 
risk  in  this  area.  Furthermore,  the  DSB  indicated  that 
significant  test  data  would  be  available  prior  to  Lot  2 
long-lead  and  contract  award  to  more  thoroughly  evaluate 
program  risks. 

The  F-22  program  will  be  conducting  full-scale  low 
observability  testing  throughout  the  engineering  and 
manufacturing  development  program.  The  testing  of  full- 
scale  hardware,  and  in  many  cases  full-scale  production 
hardware  (antennas,  leading  edges,  radome,  canopy,  etc.) 
will  significantly  reduce  the  development  risk.  Prior 
programs  with  low  observability  features  did  not  employ 
full-scale  hardware  as  is  being  done  for  the  F-22.  The  use 
of  a  radar-cross-section  measurement  system  on  the 
production  line  will  allow  for  early  evaluation  of 
production  manufacturing  processes  and  their  ability  to 
achieve  a  low  observable  aircraft . 
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The  F-22  is  the  first  aircraft  to  employ  an  integrated 
avionics  system.  One  of  the  keys  to  the  success  of  that 
system  is  the  common  integrated  processor.  In  its 
February  2,  1995,  response  to  the  referenced  GAO  report 
(OSD  Case  9494-A),  the  DoD  strongly  disagreed  with  the  GAO 
assertion  that  features  of  the  software  strategy  were  not 
being  followed.  The  DoD  pointed  out  that  the  F-22  program 
had  established  an  innovative  software  process  quality 
metric,  which  is  a  measure  of  adherence  to  all  the  processes 
identified  in  the  software  development  plans,  and  the 
associated  certification  of  those  processes  through  software 
quality  assurance  evaluations.  That  metric  currently  shows 
program  compliance  at  over  92  percent,  well  above  the 
established  target  level. 

Furthermore,  the  maturation  of  the  system/ software 
engineering  environment  (S/SEE)  was  accomplished  early  in 
the  engineering  and  manufacturing  development  phase  of  the 
program.  The  S/SEE  baseline  has  been  established  and 
currently  provides  all  the  functionality  required  for 
software  development  and  management.  Progress  to  date 
indicates  that  the  common  integrated  processor  (CIP)  is  not 
a  critical  technical  risk  area.  Preliminary  design  of  all 
of  the  avionics  applications  software  has  been  completed, 
providing  estimates  that  indicate  the  resultant  software 
will  easily  be  accommodated  within  the  data  processing 
assets  in  the  two  baseline  CIPs.  There  is  currently  a  30 
percent  management  reserve  in  both  memory  and  throughput 
without  adding  any  additional  modules. 

The  F-22  program  has  established  a  disciplined  and 
comprehensive  software  development  process.  The  DoD  is 
keenly  aware  that  the  continued  success  of  the  program  is 
contingent  on  maintaining  that  approach. 


***** 

RECOMMENDATIONS 


RECOMMENDATION  1:  The  GAO  recommended  that  the  Secretary 

of  Defense  reduce  the  degree  of  concurrency  in  the  program 
because  (1)  the  need  for  the  F-22  is  not  urgent,  (2)  the 
percentage  of  planned  F-22s  to  be  committed  to  production 
before  completion  of  lOTSE  is  higher  than  most  prior  fighter 
programs,  and  (3)  independent  testing  of  technology  advances 
will  not  be  completed  before  decisions  are  made  to  produce 
F-22S.  (pp.  20-21/GAO  Draft  Report) 
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POD  RESPONSE:  Nonconcur.  As  indicated  in  the  April  17, 
1990,  DoD  letter  to  the  Congress,  completely  sequential 
development  of  programs,  i.e.,  no  concurrency,  is 
unacceptable  because  of  time  and  cost  penalties.  The  DoD 
focus  is  on  determining  how  much  concurrency  makes  sense  in 
a  program,  and  properly  balancing  the  costs  and  risks,  both 
of  which  require  a  detailed  technical  assessment. 

The  December  1994  independent  DSB  assessment  of  the  program 
concluded  that  there  was  no  reason,  based  upon  risk  and/or 
concurrency,  to  introduce  a  schedule  stretch  in  the  F-22 
program.  The  DSB  noted  that  while  stretching  a  program  may 
reduce  risk,  if  stretching  is  not  necessary  it  can  result  in 
increased  total  costs,  possible  loss  of  key  manpower  and 
suppliers,  and  earlier  technical/operational  obsolescence. 
The  DSB  further  indicated  that  there  were  sufficient  key 
events  planned  for  each  critical  technical  area  prior  to  the 
Lot  2  long-lead  and  contract  award  for  production  which  can 
be  used  to  measure  progress.  If  there  are  significant 
delays  in  accomplishment  of  those  key  events,  or  if 
performance  levels  achieved  are  unacceptable,  the  program 
can  be  adjusted. 


RECOttffiNDATION  2;  To  minimize  commitments  to  production  of 
F-22s  until  after  successful  completion  of  lOTsE,  the  GAO 
recommended  that  the  Secretary  of  Defense  limit  LRIP 
quantities  to  that  which  can  be  produced  using  the  first  set 
of  hard  tooling — i.e.,  about  6  to  8  aircraft  per  year. 

(p.  21/GAO  Draft  Report) 


DoP  RESPONSE;  Nonconcur.  Delaying  increases  in  rate 
production  until  after  completion  of  dedicated  lOT&E  means 
that  the  high  rate  decision  (Milestone  III)  would  be  made 
prior  to  demonstrating  rate  increase  ability,  or  4-5  years 
after  dedicated  lOT&E.  Either  approach  practically  assures 
additional  or  continued  dedicated  IOT4E-like  testing  to 
confirm  production  quality/ability  before  going  into  planned 
high  rates.  The  costs  for  this  stretch-out  testing/low  rate 
production  process  can  be  expected  to  be  large  (due  to 
production  loss-of-learning,  inflation,  and  further 
engineering  and  manufacturing  development  stretch-out) .  The 
GfiO  report  omits  those  consequences  of  their  recommendation. 

Nearly  60  percent  of  the  F-22  program  is  subcontracted  to 
outside  suppliers.  The  low  rates  suggested  by  the  GAO  are 
so  unattractive  due  to  their  low  profitability  that  many 
current  suppliers  would  likely  be  lost.  In  that  case,  the 
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program  would  either  have  to  pay  high  prices  to  maintain  the 
current  supplier  base,  or  would  need  to  select  and  re¬ 
qualify  new  suppliers,  again  increasing  costs.  In  addition, 
it  is  likely  that  continued  dedicated  lOTtE  testing  would  be 
required  prior  to  approval  of  high  rate  production. 
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The  following  are  gao’s  comments  on  the  Department  of  Defense’s  (dod) 
letter  dated  February  24, 1995. 


GAO  Comments 


1.  These  comments  are  dealt  with  on  pages  12  and  13  of  the  report  and  in 
our  responses  to  the  dod  specific  comments  that  follow. 

2.  For  the  most  part,  the  risk/concurrency  guidehnes  listed  in  the  Office  of 
the  Secretary  of  Defense’s  April  1990  guide  are  specific  requirements  that 
should  be  met  before  a  program  progresses.  We  are  aware  of  many  of 
those  requirements  that  are  incorporated  in  the  F-22  program.  However, 
the  only  assessment  provided  for  in  the  guide  for  measuring  the  degree  of 
conciurency  is  the  amount  of  initial  operational  testing  and  evaluation 
(iot&e)  completed  at  the  time  low-rate  initial  production  (lrip)  begins.  By 
that  measiure,  the  F-22  program  clearly  has  a  high  degree  of  concurrency. 
In  our  opinion,  the  ramp  up  of  production  fi^om  4  a  year  to  36  a  year  imder 
the  LRIP  phase,  and  initiation  of  long  lead  for  48  a  year  essentially 
represents  a  plan  to  achieve  a  fuU-rate  production  schedule  (now  defined 
as  48  a  year)  before  iot&e  is  completed. 

3.  The  F-22  program,  as  currently  planned,  schedules  procurement  of  80 
LRIP  aircraft  at  an  estimated  cost  of  $12.4  biUion.  We  believe  that  exceeds 
the  minimum  needed  to  successfully  complete  the  lrip  phase  of  the 
program  and  that  the  production  rates  should  be  restricted  during  lrip. 
Although  many  important  F-22  development  tests  are  scheduled  prior  to 
the  acceleration  of  production  rates,  many  other  critical  developmental 
tests  and  most  iot&e  testing  are  not  scheduled  to  be  complete  until  after 
significant  commitments  are  made  to  production. 

4.  We  adjusted  the  report  to  reflect  this  information.  However,  it  should  be 
noted  that  the  total  number  of  each  type  of  aircraft  produced  was  much 
higher  than  planned  for  the  F-22.  This  results  in  a  higher  degree  of 
concurrency  in  the  F-22  program  when  using  the  percentage  of  aircraft 
procured  at  completion  of  iot&e  as  a  measiure  of  concurrency. 

5.  Oxu-  report  does  not,  either  explicitly  or  implicitly,  suggest  “total 
avoidance”  of  concurrency. 

6.  The  Defense  Science  Board  (dsb)  portrayal  of  the  F-22  program  as 
relatively  conservative  was  based  on  the  amount  of  development  testing  to 
be  completed  at  early  production  decision  points.  However,  using  the 
measure  called  for  by  dod’s  own  1990  guidance — the  amount  of  iot&e 
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completed  at  the  time  lrip  begins— shows  that  the  F-22  program  is  far 
from  conservative. 

7.  Production  ramp  up  from  4  aircraft  a  year  to  36  aircraft  a  year  appears 
to  provide  a  more  rapid  acceleration  than  we  believe  is  necessary  in  the 
LRIP  phase  of  the  program.  In  our  opinion  the  ramp  up  of  production  from 
4  a  year  to  36  a  year  under  the  lrip  phase,  and  initiation  of  long  lead  to 
support  48  a  year,  essentially  represents  a  plan  to  achieve  a  full-rate 
production  schedule  before  iot&e  is  completed. 

8.  This  material  has  been  deleted  from  the  report. 

9.  Additional  information  concerning  this  matter  has  been  added  to  the 
body  of  the  report. 

10.  DOD  response  to  our  prior  report  on  embedded  computers  has  been 
recognized  in  the  body  of  this  report. 

1 1.  We  did  not  attempt  to  quantify  potential  cost  growth  in  the  F-22 
program  that  may  result  from  a  change  in  the  program  schedule.  However, 
the  thrust  of  the  lrip  legislation  is  to  authorize  only  minimum  necessary 
quantities.  DOD  acquisition  profiles  created  for  other  weapon  programs 
have  often  proven  to  be  optimistic  and  are  rarely  carried  out  as  initially 
planned  because  of  technical,  financial,  or  test  problems.  If  the  baseline 
against  which  to  compare  potential  growth  of  costs  is  optimistic,  an 
estimate  of  cost  growth  would  have  limited  meaning  at  this  point  because 
those  problems  are  likely  to  occur  in  highly  concurrent  programs  that 
involve  substantial  advances  in  technology. 
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